perf(memtrack): reduce serialization bottleneck by not-matthias · Pull Request #436 · CodSpeedHQ/codspeed

not-matthias · 2026-07-06T15:04:34Z

Summary

replace derived flattened MemtrackEvent serialization with a byte-identical manual serializer
make MemtrackWriter::finish return encoded frame bytes for batching
encode contiguous memtrack event batches on worker threads and write sequence-numbered zstd frames in order

Verification

cargo test -p runner-shared artifacts::memtrack
cargo test -p runner-shared
cargo check -p runner-shared
cargo test --manifest-path crates/memtrack/Cargo.toml via Nix shell with libclang/build deps; eBPF integration cases compiled but were ignored because GITHUB_ACTIONS was unset
cargo bench -p runner-shared --bench memtrack_writer

Notes

memtrack_writer bench measures the Phase A single-writer encoder path only, not the full parallel memtrack pipeline.
End-to-end memory-mode speedup was not measured in this environment.

codspeed-hq · 2026-07-06T15:10:46Z

Merging this PR will not alter performance

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks

_{Comparing cod-3071-fix-serialization-bottleneck (64cd996) with main (5c27231)¹}

No successful run was found on main (4da30e9) during the generation of this report, so 5c27231 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

greptile-apps · 2026-07-06T15:16:43Z

Greptile Summary

This PR replaces the single-threaded, event-at-a-time memtrack write path with a parallel pipeline: events are batched into 64K-event windows, each window is divided into fixed-size zstd frames that are compressed concurrently by a Rayon pool, and the frames are written to disk in input order. It also switches the BPF ring buffer poller from std::sync::mpsc to crossbeam_channel, adds a watermark-based wakeup scheme to the eBPF side to reduce per-event syscall overhead, and expands the ring buffer from 16 MB to 256 MB.

Parallel encoding pipeline (pipeline.rs): encode_events consumes any IntoIterator<Item = MemtrackEvent>, chunks it into windows of up to 1M events, compresses 64K-event frames in parallel via a dedicated Rayon pool, and writes them in order.
Writer refactor (writer.rs): MemtrackWriter::finish() now returns the inner sink; BufWriter and zstd encoder layers are reordered, and the compression level is dropped from 1 to -5 for throughput.
Poller / tracker cleanup: RingBufferPoller is now owned by Tracker and torn down via stop_polling(), eliminating the forwarding thread; a final consume() after the shutdown loop ensures tail events reach the channel before tx is dropped.

Confidence Score: 5/5

The PR is safe to merge. The new pipeline correctly preserves event order, bounds peak memory, and handles shutdown cleanly.

The core encoding, ordering, and shutdown paths were all traced end-to-end and are correct. Tests cover byte-identity of serialisation, order preservation across parallel frames, window boundaries, and the empty-stream edge case. The only findings are a stale comment on compression level and thread-pool creation inside the benchmark loop.

No files require special attention; the benchmark file has a minor measurement-noise issue worth a second glance.

Important Files Changed

Filename	Overview
crates/runner-shared/src/artifacts/memtrack/pipeline.rs	New encode pipeline: chunks events into 64K-event frames, compresses frames in parallel via a dedicated Rayon pool, writes them in order. Logic, ordering, memory bounds, and error handling are all correct.
crates/runner-shared/src/artifacts/memtrack/writer.rs	Refactored MemtrackWriter to return the inner sink from finish(), enabling frame-byte extraction. BufWriter/zstd layering was swapped; compression level dropped from 1 to -5 for throughput. The comment is now stale.
crates/runner-shared/src/artifacts/memtrack/mod.rs	Reorganised into submodules; test coverage added for byte-identity of serialisation and concatenated-frame decoding. No issues found.
crates/memtrack/src/main.rs	Replaced manual drain/writer threads with a single encode_events pipeline_thread; shutdown sequence is correct and race-free.
crates/memtrack/src/ebpf/poller.rs	Switched from std::sync::mpsc to crossbeam_channel::unbounded; final consume() after shutdown loop ensures tail events reach the channel before tx is dropped.
crates/memtrack/src/ebpf/tracker.rs	Added stop_polling() which drops the RingBufferPoller triggering shutdown and channel close; poller is now owned by the Tracker.
crates/memtrack/src/ebpf/c/memtrack.bpf.c	Ring buffer size increased 16x to 256MB; per-event wakeup replaced with watermark-based scheme. Logic is sound.
crates/runner-shared/benches/memtrack_writer.rs	New encode_events_realistic bench measures full 1M-event pipeline. Rayon thread pool is created and destroyed inside each bench iteration, adding thread-spawn overhead to each measurement.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant eBPF as eBPF Kernel
    participant Poller as RingBufferPoller
    participant Channel as crossbeam channel
    participant Pipeline as encode_events
    participant Rayon as Rayon Pool
    participant Disk as Output File
    eBPF->>Poller: ring buffer events
    loop poll loop 10ms timeout
        Poller->>Poller: ringbuf.poll then consume
        Poller->>Channel: tx.send(event)
    end
    Channel->>Pipeline: event_rx iterator
    loop window up to 1M events
        Pipeline->>Rayon: par_chunks(64K).map(encode_frame)
        Rayon-->>Pipeline: frames in order
        Pipeline->>Disk: write frames in order
    end
    note over Poller: stop_polling drops Poller closes channel
    Pipeline-->>Pipeline: exhausted flush return total

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant eBPF as eBPF Kernel
    participant Poller as RingBufferPoller
    participant Channel as crossbeam channel
    participant Pipeline as encode_events
    participant Rayon as Rayon Pool
    participant Disk as Output File
    eBPF->>Poller: ring buffer events
    loop poll loop 10ms timeout
        Poller->>Poller: ringbuf.poll then consume
        Poller->>Channel: tx.send(event)
    end
    Channel->>Pipeline: event_rx iterator
    loop window up to 1M events
        Pipeline->>Rayon: par_chunks(64K).map(encode_frame)
        Rayon-->>Pipeline: frames in order
        Pipeline->>Disk: write frames in order
    end
    note over Poller: stop_polling drops Poller closes channel
    Pipeline-->>Pipeline: exhausted flush return total

_{Reviews (13): Last reviewed commit: "perf(memtrack): drain ringbuf bursts eag..." | Re-trigger Greptile}

GuillaumeLagrange

lgtm

…t zstd compression bottleneck

Callers that encode a frame into an in-memory buffer need the buffer back after the zstd stream is finalized, so frames can be composed into a larger artifact stream. Refs COD-3071

Group events into fixed-size frames and compress them across a rayon pool, writing windows in input order. Bounds peak memory regardless of run length and removes the single-threaded compression bottleneck. Refs COD-3071

…race Generate a seeded malloc/free/realloc/mmap workload with a live-heap model instead of uniform random events, and sweep worker counts. Refs COD-3071

Forward ring buffer events over a crossbeam channel directly instead of through an extra keep-alive forwarding thread. On shutdown, consume the ring buffer once more so events emitted after the last poll are not lost, then close the channel. Refs COD-3071

Feed the poller's event channel straight into encode_events on one thread, sized to available parallelism. Shutdown is now deterministic: stop the poller, which drains stragglers and closes the channel, then join the pipeline. Fixes COD-3071

itertools::chunks erases the iterator size hint, so collecting each window grew by doubling (~29% of encode driver time in reallocation). A single pre-sized buffer reused across windows removes the growth and the chunk-adapter bookkeeping: encode_events_realistic median drops 21%/17%/9% at 16/8/4 workers.

Submitting with default flags wakes the consumer on nearly every event, and the wakeup dominates submission cost at high event rates. Submit with BPF_RB_NO_WAKEUP until 64KB of unconsumed data has accumulated, then force a wakeup. Kernel ringbuf benchmarks show up to ~20x higher sustained throughput with sampled notification. The poller's 10ms poll timeout flushes the tail that never reaches the watermark. Refs COD-3071

16 MiB holds only ~300K records (~60ms of headroom at multi-M events/s bursts), so a short consumer scheduling stall overflows the buffer and drops events. 256 MiB gives ~1s of worst-case burst headroom. The buffer is memcg-charged and vmap'd, so the only cost is the memory itself while tracking runs. Refs COD-3071

Two consumer-side fixes for event drops under allocation bursts: - After each poll() the poller calls consume() once, which drains the ring buffer to empty without paying an epoll wakeup round-trip per poll cycle. - The encode pipeline no longer claims every core: rayon workers on all cores starve the poll thread (and the tracked command), which is what let the ring buffer fill up. Reserve two cores. Refs COD-3071

not-matthias force-pushed the cod-3071-fix-serialization-bottleneck branch from 381bac8 to 9be9b22 Compare July 6, 2026 15:10

greptile-apps Bot reviewed Jul 6, 2026

View reviewed changes

Comment thread crates/memtrack/src/main.rs Outdated

Comment thread crates/memtrack/Cargo.toml Outdated

Comment thread crates/memtrack/src/main.rs Outdated

not-matthias force-pushed the cod-3071-fix-serialization-bottleneck branch 2 times, most recently from a225638 to 9561cdd Compare July 7, 2026 17:43

greptile-apps Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread crates/runner-shared/src/artifacts/memtrack/pipeline.rs Outdated

not-matthias force-pushed the cod-3071-fix-serialization-bottleneck branch from 9561cdd to 9292b3f Compare July 7, 2026 18:00

not-matthias marked this pull request as ready for review July 7, 2026 18:19

not-matthias force-pushed the cod-3071-fix-serialization-bottleneck branch from 346a162 to 99b9b68 Compare July 14, 2026 10:05

not-matthias requested a review from GuillaumeLagrange July 14, 2026 10:05

GuillaumeLagrange approved these changes Jul 15, 2026

View reviewed changes

not-matthias added 9 commits July 15, 2026 19:08

refactor: split memtrack into module

21ba525

fix: wire MemtrackWriter serializer through BufWriter to fix per-even…

aadcfb8

…t zstd compression bottleneck

fix: lower zstd compression level to -5 to reduce serialization overhead

1a1ab29

ref(memtrack): return the underlying sink from MemtrackWriter::finish

68b9eeb

Callers that encode a frame into an in-memory buffer need the buffer back after the zstd stream is finalized, so frames can be composed into a larger artifact stream. Refs COD-3071

test(memtrack): benchmark encode_events with a realistic allocation t…

efa5d31

…race Generate a seeded malloc/free/realloc/mmap workload with a live-heap model instead of uniform random events, and sweep worker counts. Refs COD-3071

not-matthias force-pushed the cod-3071-fix-serialization-bottleneck branch from 99b9b68 to 78d5ae2 Compare July 15, 2026 17:17

not-matthias added 4 commits July 15, 2026 19:24

ci: also run runner-shared benchmarks in walltime mode

88a6136

ci: install rust toolchain by explicit channel for older rustup

d06d42e

greptile-apps Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread crates/memtrack/src/ebpf/poller.rs Outdated

not-matthias force-pushed the cod-3071-fix-serialization-bottleneck branch from 64cd996 to 5c27231 Compare July 16, 2026 15:10

not-matthias merged commit 5c27231 into main Jul 16, 2026
22 checks passed

not-matthias deleted the cod-3071-fix-serialization-bottleneck branch July 16, 2026 15:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(memtrack): reduce serialization bottleneck#436

perf(memtrack): reduce serialization bottleneck#436
not-matthias merged 14 commits into
mainfrom
cod-3071-fix-serialization-bottleneck

not-matthias commented Jul 6, 2026

Uh oh!

codspeed-hq Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 6, 2026 •

edited

Loading

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

not-matthias commented Jul 6, 2026

Summary

Verification

Notes

Uh oh!

codspeed-hq Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

greptile-apps Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq Bot commented Jul 6, 2026 •

edited

Loading

greptile-apps Bot commented Jul 6, 2026 •

edited

Loading